UCYMICRA: Distributed Indexing of the Web Using Migrating Crawlers
نویسندگان
چکیده
Due to the tremendous increase rate and the high change frequency of Web documents, maintaining an up-to-date index for searching purposes (search engines) is becoming a challenge. The traditional crawling methods are no longer able to catch up with the constantly updating and growing Web. Realizing the problem, in this paper we suggest an alternative distributed crawling method with the use of mobile agents. Our goal is a scalable crawling scheme that minimizes network utilization, keeps up with document changes, employs time realization, and is easily upgradeable.
منابع مشابه
Distributed Indexing of the Web Using Migrating Crawlers
Due to the tremendous increase rate and the high change frequency of Web documents, maintaining an up-to-date index for searching purposes (search engines) is becoming a challenge. The traditional crawling methods are no longer able to catch up with the constantly updating and growing Web. Realizing the problem, in this paper we suggest an alternative distributed crawling method with the use of...
متن کاملReducing Network Traffic and Managing Volatile Web Contents Using Migrating Crawlers with Table of Variable Information
As the size of the web continues to grow, searching it for useful information has become increasingly difficult. Also study reports that sufficient of current internet traffic and bandwidth consumption are due to the web crawlers that retrieve pages for indexing by the different search engines. Moreover, due to the dynamic nature of the web, it becomes very difficult for a search engine to prov...
متن کاملUsing Migrating Agents in Designing Web Search Engines and Property Analysis of Available Platforms
Mobile agent is a type of software agent, a composition of computer software and data. It is able to migrate from one computer to another autonomously and continue its execution on the destination computer. When very large volumes of data are stored at remote hosts, these data can be processed in the locality of the data rather than transferring over the network. The main concern is, move the c...
متن کاملDesign and Implementation of a High-Performance Distributed Web Crawler
Broad web search engines as well as many more specialized search tools rely on web crawlers to acquire large collections of pages for indexing and analysis. Such a web crawler may interact with millions of hosts over a period of weeks or months, and thus issues of robustness, flexibility, and manageability are of major importance. In addition, I/O performance, network resources, and OS limits m...
متن کاملNeed of Securing Migrating Crawling Agent, Remote Platform and the Data Collection
Using migrating (mobile) crawling agents, the process of selection and filtration of web documents can be done at web servers rather than search engine side, which reduces network load caused by the web crawlers. The mobile code from search engine side transfers and executes on web servers, an environment controlled by another party, it gives rise to several security issues in mobile agent comp...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2003